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Abstract 


New  vision  applications  have  been  made  possible  and  old  ones  improved  through  the  creation  and 
design  of  novel  catadioptric  systems.  Critical  to  the  design  of  catadioptric  imaging  is  determining 
the  shape  of  one  or  more  mirrors  in  the  system.  Almost  all  the  previously  designed  mirrors  for 
catadioptric  systems  used  case  specific  tools  and  considerable  effort  on  the  part  of  the  designer. 
Recently  some  new  general  methods  have  been  proposed  to  automate  the  design  process.  However, 
all  the  methods  presented  so  far  determine  the  mirror  shape  by  optimizing  its  geometric  properties, 
such  as  surface  normal  orientations.  A  more  principled  approach  is  to  determine  a  mirror  that 
reduces  image  errors. 

In  this  paper  we  present  a  method  for  finding  mirror  shapes  which  meet  user  determined  specifi¬ 
cations  while  minimizing  image  error.  We  accomplish  this  by  deriving  a  first  order  approximation 
of  the  image  error.  This  permits  us  to  compute  the  mirror  shape  using  a  linear  approach  that 
provides  good  results  efficiently  while  avoiding  the  numerical  problems  associated  with  non-linear 
optimization.  Since  the  design  of  mirrors  can  also  be  applied  to  projection  systems,  we  also 
provide  a  method  to  approximate  projection  errors  in  the  scene.  We  demonstrate  our  approach 
on  various  catadioptric  systems  and  show  our  approach  to  provide  much  more  accurate  imaging 
characteristics.  In  some  cases  we  achieved  reduction  in  image  error  upto  80  percent. 
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1  Design  of  Catadioptric  Systems 


The  capability  of  a  vision  system  is  largely  dependent  on  the  type  of  imaging  system  it  uses. 
For  instance,  creating  360  deg  panoramic  video  with  a  single  conventional  (perspective)  camera  is 
impossible.  It  can  easily  be  accomplished,  however,  with  specially  designed  optics.  Recognizing 
this  fact,  many  unconventional  imaging  systems  which  use  both  lenses  and  mirrors  (catadioptric 
systems)  have  been  proposed  in  recent  years. 

A  common  use  of  catadioptric  imaging  is  to  enhance  the  held  of  view.  Systems  that  achieve  this 
goal  have  been  developed  using  both  planar  as  well  as  curved  mirrors.  Some  of  these  systems  have 
been  designed  to  have  a  single  effective  viewpoint  [20,  26,  27,  18,  19,  l].  Other  systems  have  been 
designed  to  achieve  specific  resolution  characteristics,  such  as  the  equi- angular  mapping  [6],  the 
constant-resolution  mapping  [9]  and  equi-areal  mapping  [16].  Systems  have  also  been  designed 
to  acquire  images  of  a  known  scene  in  special  ways[3,  7,  2,  14,  8,  5,  17,  13]. 

A  critical  aspect  of  designing  a  catadioptric  imaging  system  is  determining  the  appropriate  shapes 
of  the  one  or  more  mirrors  that  the  system  uses.  In  general,  the  performance  of  an  imaging  system 
has  two  components.  One  is  its  ability  to  adhere  to  the  prescribed  image-to-scene  map  and 
the  second  is  its  optical  image  quality  (determined  by  depth  of  held,  coma,  astigmatism,  etc.  [4, 
12]).  This  paper  focuses  on  the  first  of  these  components. 

Let  us  consider  a  system  that  has  a  single  mirror.  To  determine  the  shape  of  this  mirror,  the 
primary  optics  (perspective  lens,  telecentric  lens  or  otherwise)  and  the  image-to-scene  map  (a 
map  from  pixels  in  the  image  to  scene  points  or  viewing  directions)  need  to  be  known.  Hicks  and 
Perline  [15]  were  the  first  to  pose  the  problem  for  mirror  design  using  “geometric  distributions”. 

The  mirror  for  each  of  the  systems  designed  previously  was  found  by  using  the  constraints  imposed 
by  the  image-to-scene  map  to  derive  partial  differential  equations  (PDEs)  that  the  mirror  must 
satisfy.  The  mirror  shape  was  then  determined  by  finding  solutions  to  the  PDEs.  This  approach 
requires  considerable  skill  and  effort  on  the  part  of  the  designer.  A  more  general  approach  using 
splines  to  model  and  compute  the  mirror  shape  was  recently  presented  in  [25].  This  method  is 
similar  to  an  earlier  method  developed  in  [ll]  to  recover  the  shape  of  a  specular  surface  given 
known  scene  points  and  a  perspective  camera. 

An  important  problem  that  arises  in  designing  mirrors  is  that  there  may  not  exist  a  single  mirror 
that  can  exactly  achieve  the  required  image-to-scene  map.  In  such  cases,  all  the  mirror  design 
methods  described  above  determine  the  “best-fit”  mirror  surface  that  minimizes  errors  in  its  three 
dimensional  shape  (surface  normals  of  the  mirror).  However,  as  shown  in  Fig.l,  a  mirror  that 
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Scene 


Figure  1 :  Two  points  on  a  mirror  of  a  catadioptric  system  reflecting  rays  towards  the  scene  are  perturbed 
by  the  same  small  angle  9.  Clearly  the  two  equal  perturbations  cause  unequal  errors  on  the  scene 
projection.  Most  mirror  design  methods  try  to  minimize  errors  in  the  shape  of  the  mirror  itself  (such  as 
surface  normal  orientation)  without  regard  to  its  effects  on  the  scene  or  the  acquired  image.  Thus,  the 
image  acquired  with  such  a  mirror  is  not  guaranteed  to  exhibit  minimum  image  error. 

minimizes  errors  in  its  normals  need  not  minimize  image  errors.  That  is,  the  resulting  mirror  may 
have  normals  that  are  close  to  the  desired  set  of  normals  but  produces  very  large  distortions  in 
the  captured  image.  Since  we  are  interested  in  the  geometric  quality  of  the  image  captured  by 
the  imaging  system,  the  correct  approach  is  to  find  a  mirror  that  minimizes  image  errors. 

Note  that  a  method  for  designing  mirrors  for  imaging  can  also  be  used  to  design  projection 
systems.  In  this  case,  the  mirror  must  minimize  scene  projection  errors  instead.  In  general, 
minimizing  either  the  image  error  or  scene  error  requires  a  non-linear  optimization,  which  can 
be  highly  unstable.  In  [25],  the  authors  recognized  this  problem  and  addressed  it  by  using  a 
heuristic  weighting  approach.  However,  this  technique  does  not  accurately  model  either  image  or 
scene  errors  and  hence  does  not  guarantee  good  solutions. 

In  this  work,  we  derive  first-order  approximations  to  image  and  scene  errors.  We  incorporate  these 
approximations  into  the  algorithm  in  [25]  to  determine  the  mirror  shape  that  minimizes  image 
or  scene  errors,  without  the  need  for  a  costly  non-linear  optimization.  We  present  experimental 
results  to  demonstrate  the  effectiveness  of  such  an  approach. 


2  Spline-Based  Mirror  Design 

The  catadioptric  system  to  be  designed  is  assumed  to  consist  of  some  known  primary  optics  and 
an  unknown  mirror.  The  primary  optics  are  assumed  to  be  fixed  and  known.  The  designer  of 
the  imaging  or  projection  system  presents  the  requirements  of  the  system  in  terms  of  a  map  from 
pixels  in  the  image  to  points  in  the  scene.  We  call  this  the  image-to-scene  map  which  we  denote 
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Figure  2:  The  different  models  of  the  primary  optics  for  which  our  method  can  compute  a  mirror  shape, 
(a)  A  perspective  imaging  model  found  in  most  projectors  and  imaging  systems,  (b)  Orthographic 
projection,  typically  obtained  with  telecentric  lenses,  (c)  The  generalized  imaging  model  (caustic  model) 
,  wherein  each  pixel  can  have  its  own  associated  viewpoint  and  viewing  direction.  This  last  model  is 
most  general  and  allows  the  most  flexibility  in  designing  catadioptric  systems. 

A4(u,  v).  Our  goal  is  the  find  the  mirror  which  along  with  the  known  primary  optics  implements 
the  desired  image-to-scene  map  Ai. 

In  this  paper  we  use  a  previously  proposed  method  for  general  imaging  system  design  using  splines 
[25].  Our  emphasis  in  this  work  however,  is  to  modify  the  equations  used  in  previous  work  so 
that  the  mirror  we  compute  minimizes  image  or  scene  errors.  The  previous  approach  based  on  a 
specular  shape  recovery  technique  [ll]  essentially  determines  the  mirror  by  minimizing  errors  in 
its  surface  normal  orientations.  Before  we  present  the  details  of  our  approach,  we  shall  introduce 
certain  notation  from  previous  work  which  will  be  useful.  Throughout  this  paper  we  will  address 
the  problem  of  catadioptric  imaging  system  design.  However,  the  same  analysis  can  be  directly 
applied  to  projector  design  as  well. 

2.1  Modeling  the  Primary  Optics 

The  primary  optics  of  the  catadioptric  system  could  either  use  a  conventional  lens  (perspective 
projection)  as  in  Fig.  2(a),  or  a  telecentric  lens  (orthographic  projection)  as  shown  in  Fig.  2(b), 
or  the  generalized  model[  10,  23]  (see  Fig.  2(c)).  Referring  to  Fig.  2(c)),  in  general  a  pixel  (u,  v)  in 
the  image  possesses  a  viewpoint  Si(u,u)  and  a  viewing  direction  Vi(w,u).  For  a  perspective  lens, 
the  viewpoints  Si(u,  v )  collapse  to  a  single  point. 
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2.2  Modeling  the  Mirror  Shape 


We  describe  the  mirror  shape  Sr  in  terms  of  the  model  for  the  primary  optics  as: 

Sr(u,  v)  =  Si(u,  v)  -  D{u ,  v)Vi(u,  v ),  (1) 

where  D(u,  v )  is  the  distance  of  the  mirror  from  the  viewpoint  surface,  modeled  using  the  tensor 
product  spline : 

Kf  Kg 

D(u ,  v)  =  £  Cijfi(u)gj(v),  (2) 

i=l  j= 1 

where  ft(v)  and  gj(v)  are  1-D  spline  basis  functions,  ch:j  are  the  coefficients  of  the  spline  model, 
and  Kf  ■  Kg  are  the  number  of  spline  coefficients. 


2.3  Mirror  Normal  Constraints 


Fig.  3  shows  a  catadioptric  system  used  to  image  some  known  scene.  The  mirror  surface  Sr(it,  v ) 
must  implement  the  user  provided  image  to  scene  mapping  A4  by  reflecting  each  scene  point 
A4(u,  v )  along  the  scene  ray  Vr(«,  v)  into  the  primary  optics,  where: 

V  (y  v)  =  Sr {U,V)-M(U,V) 

rV  '  ’  |Sr(«,u)  -M{u,v)\' 

This  constrains  the  surface  normal  of  the  mirror  Nr  as: 


Nr(«,  v) 


Vi (u,v)  -  Vr(«d) 

Vi(w,u)  -  Vr(n,n)|' 


(4) 


Figure  3:  A  catadioptric  imaging  system  consisting  of  some  known  primary  optics  and  a  mirror.  In 
general,  a  pixel  (u,v)  in  the  image  maps  to  the  scene  point  M(u,v)  after  reflecting  at  Sr(u,v )  on  the 
mirror.  This  forces  constraints  on  the  surface  normals  N r(u,v)  of  the  mirror. 
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The  tangent  vectors  T U(u,v)  =  9Svq ^  and  T„(n,n)  =  dSr^’v^  on  the  mirror  surface  must  be 
orthogonal  to  the  surface  normal  in  Eq.(4).  This  provides  two  constraints  on  the  mirror  shape: 


Tu(u,v)  ■  Nr(u,v)  =  0, 


(5) 


Tv(u,  v)  •  Nr(«,  v)  =  0. 


Rearranging  the  terms  and  substituting  Eq.(l)  in  Eq.(5),  we  get: 

(^V,  +  Z^-^VNr  =  0, 

V  ou  ou  ou  ) 


-  fl)  •  N,  =  0. 

\  ov  ov  ov  J 

Now,  substituting  Eq.(2)  in  Eq.(7),  we  get: 


V1  E  Ci,jfi(U)9j(v)  +  ^  E  Ci,jfi(U)9j(v)  -  ^ 

't'lJ  't'lJ 


•  Nr  =  0 


cf\l  r)  S 

V1  E  ci,jfi(u)9j(v)  +  E  ci,jfi(u)9j(v)  -  ]  •  Nr  =  0. 


h3 


hJ 


dv 


(6) 


(7) 


where  f[{u)  and  g'j(v)  denote  the  derivatives  of  fi(u)  and  gj(v),  respectively.  Note  that  the  above 
constraints  quantify  the  error  in  mirror  shape  in  terms  of  the  dot  product  error  between  the 
tangents  and  the  desired  surface  normals  of  the  mirror.  Furthermore  they  are  linear  in  the  spline 
coefficients  Cij. 


3  Reducing  Scene  and  Image  Error 

We  now  present  a  simple  technique  to  reformulate  the  constraints  in  Eqs.  (5-8)  so  that  solutions 
to  the  modified  equations  will  approximately  minimize  image  or  scene  errors.  Before  doing  so 
however,  we  need  to  define  scene  error  and  image  errors. 

Consider  a  catadioptric  system  that  must  implement  some  image-to-scene  map  A4  (u,v).  When 
no  mirror  shape  exists  that  can  exactly  implement  A 4(u,v),  the  mirror  shape  is  approximate. 
Therefore  a  light  ray  corresponding  to  pixel  (u,  v)  gets  reflected  towards  a  scene  point  other  than 
the  required  point  A4(u,v).  Let  rj(u,v)  be  the  effective  (possibly  erroneous)  image-to-scene  map 


6 


that  the  catadioptric  system  with  a  given  mirror  implements.  The  scene  error  £p  at  a  point  (u,v) 
is  then  given  by: 

Zp  =  \v(u,v)  -  M(u,v)\.  (8) 

For  an  imaging  system,  the  image  error  can  be  computed  by  simply  inverse  mapping  the  effective 
scene  point  rj(u,v)  onto  the  image  using  the  ideal  image-to-scene  map  A4  (u,v).  The  image  error 
at  pixel  (u,  v)  is: 

£/  =  |Af_1(r7(n,n))  -  {u,v)\.  (9) 


When  an  exact  mirror  shape  exists,  the  errors  £/,  £ p  and  Eqs.(5)  all  vanish.  When  no  mirror  shape 
exists  that  implements  the  desired  image-to-scene  map,  the  Eqs.  (5)  are  not  satisfied  and  can  be 


rewritten  as: 


(  e°’“  \  =  (  Tu  ‘  Nr 
V  Zo,v  )  \Tv^r 


(10) 


where  £o,«  and  £o,t>  denote  the  residues  corresponding  to  the  two  constraint  equations  respectively. 


The  least  squares  solution  of  Eqs.  (5)  minimizes  the  mean  square  of  the  residual  £o-  Our  goal  is 
to  formulate  linear  equations  whose  least  square  solutions  better  approximate  the  image  or  scene 
errors.  To  do  so,  we  first  derive  a  relationship  between  the  residual  and  angular  perturbations  of 
the  tangent  vectors.  Next  we  relate  the  angular  tangent  perturbations  to  errors  in  the  reflected 
rays  and  thus  scene  errors.  The  last  step  is  to  relate  the  scene  errors  to  image  errors. 


3.1  Relating  Angular  Perturbations  of  Tangents  to  Residuals 


Referring  to  Eq.  (1)  for  the  mirror  shape,  the  surface  tangents  are  given  by  T u  =  ^  and  T„  =  ^ 
respectively.  Taking  tangent  T„  into  consideration,  we  first  note  that  it  can  be  rewritten  as: 


T 


dD 

du 


Vi  + 


dSi 

du 


(11) 


For  a  given  pixel  (u,v),  the  quantities  and  ^  depend  on  the  primary  optics  and  are  inde¬ 
pendent  of  the  mirror  shape  (tensor-spline  coefficients).  In  analyzing  the  perturbations  of  the 
tangents  we  consider  the  location  of  a  point  on  the  mirror  Sr(u,u)  to  be  essentially  fixed.  Thus, 
D(u,  v )  changes  slowly  with  respect  to  changes  in  the  spline  coefficients.  The  variation  in  tangents 
with  respect  to  the  coefficients  is  thus  almost  entirely  due  to  changes  in  9Dq^  •  This  restricts 
to  a  plane  Vu  spanned  by  the  vectors  Vi  and  D Thus,  perturbations  to  can  be 
expressed  simply  as  a  rotation  in  this  plane. 
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Let  7ru(Nr)  be  the  projection  of  Nr  onto  the  plane  Vu-  We  note  that  the  component  of  Nr  which 
is  orthogonal  to  plane  Vu  is  automatically  orthogonal  to  Tu.  Therefore  this  component  of  Nr  is 
irrelevant.  We  can  then  express  the  angle  9U  between  7r„(Nr)  and  in  terms  of  the  residual  as: 

cos  6b  =  fo,«/(|Tu||7ru(Nr)).  (12) 

Similarly,  £ q,v  can  be  related  to  9V  as 

cos  6b  =  io,v/ (|T„||7r„(Nr)).  (13) 


The  angles  (9U,9V)  specify  the  directions  of  the  tangents  for  a  given  iteration.  We  write  the  first 
order  approximation  for  a  perturbation  of  these  angles  by  (A9U,A9V)  as: 

(  Mu  (  C°t{-6u)  )  -  L  (  )  (14) 

V  A0 V  )  V  Cot(^)  )  V  &,v  ) 

where  L  is  a  diagonal  matrix  with  diagonal  elements 

Li  =  -l/sin(0u)|T11||7rQu(NP)| 

L2  =  -l/sin(6g|T„||7rcjNr)| 


The  quantity  7rjj(Nr)  can  be  computed  at  every  iteration  of  our  method  and  does  not  depend  on 
the  unknown  tangents.  The  length  |TU|  does  depend  on  the  unknown  tangents.  However,  in  our 
iterative  method  to  determine  the  mirror  shape,  we  find  that  |TU|  remains  relative  constant  across 
iterations.  The  reasons  for  this  can  be  seen  in  Eq.  (11).  As  already  noted,  T„  is  a  sum  of  three 
terms  with  ^  constant  for  each  iteration  and  with  D changing  slowly  because  it  depends  only 
on  the  mirror  position. 

So  far  we  derived  a  first  order  relationship  between  the  residuals  of  the  constrains  of  Eqs.  (5)  and 
perturbations  of  the  tangent  vectors.  We  shall  use  this  relationship  next  to  determine  the  scene 
errors. 


3.2  Relating  Scene  Error  to  Residuals 

We  now  present  our  method  of  relating  tangent  perturbations  to  scene  errors.  This  will  be  used 
to  modify  Eqs.  (5)  in  order  to  approximate  the  scene  errors. 

The  three  dimensional  vector  which  represents  the  scene  error  is: 

=  (Cp,x>  £p,z)  =  V(U)V)  —  M{UjV).  (16) 
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For  a  given  pixel  we  consider  the  vector  scene  error  £p  as  a  function  of  the  angles  9U  and  6V.  The 
first  order  approximation  of  the  vector  scene  error  due  to  angular  perturbations  A 9U  and  A 9V  is 
then  given  by: 

f  Cp,x(9u,9v) 

Cp,y(9ut  9v) 

\  £p,z(@uj  @v) 

where  J(£p)  is  the  Jacobian,  given  by: 


+  J(£p) 


A  9„ 


(17) 


ACp) 


du 

dv 

&£p,y 

du 

dv 

d£p,z 

du 

dv 

(18) 


When  the  angles  of  the  tangent  vectors  of  the  mirror  Tu  and  T„  are  perturbed  by  a  small  amount 
A 9U  and  A 9V  about  7t/2,  the  cotangent  term  in  Eq.  (14)  vanishes  and  sin(7r/2)  =  1  in  Eq.  (15). 
Combining  these  equations  with  Eq.  (17)  we  have  the  first  order  approximation  for  scene  error  as 

£p  *  JiQLZo-  (19) 


Observe  from  Eq.  (5)  that  the  residuals  £ o  are  given  by  a  pair  of  linear  equations  in  the  spline 
coefficients  for  each  pixel.  We  multiply  each  pair  of  equations  from  the  left  by  the  3x2  matrix 
J(£P)L  to  obtain  three  linear  equations  in  the  spline  coefficients.  A  least  squares  solution  of  these 
modified  linear  equations  approximates  the  minimization  of  the  scene  error  to  first  order.  These 
are  the  equations  we  use  to  compute  the  mirror  shape  for  catadioptric  projection  systems. 


3.3  Relating  Image  Error  to  Residuals 

For  imaging  systems  we  wish  to  minimize  the  image  error  defined  in  Eq.  (9).  This  can  be  achieved 
by  mapping  the  scene  errors  back  to  the  image  using  the  inverse  At-1  of  the  image-to-scene  map. 
However,  since  the  Jacobian  of  the  image-to-scene  map  J(A4)  is  a  3  x  2  matrix,  we  employ  a  left 
pseudo-inverse  to  transform  the  scene  error  to  the  image  error  as 

£/  ~  (JiMfJiM))-1  J(M)tJ({p)£p.  (20) 

Combining  this  with  Eq.  (19),  a  first  order  approximation  for  image  error  is  then: 

£/  ~  ((JiMfJiM))-1  J(M)TJ(Cp)L^o-  (21) 
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Hence,  we  now  have  a  simple  method  to  transform  the  residuals  in  Eqs.  (5)  into  the  image  or  scene 
error.  To  do  so,  we  simply  multiply  the  constraint  equations  by  an  appropriate  transformation 
matrix  as  in  Eqs.  (19,21).  This  results  in  a  new  set  of  linear  equations  that  approximate,  to  first 
order,  the  minimization  of  the  scene  or  image  error  respectively. 


4  Computing  the  Mirror  Shape 

We  now  present  an  iterative  algorithm  to  compute  the  mirror  shape  for  a  desired  catadioptric 
system  using  the  modified  constraints  presented  in  the  previous  section. 

We  first  note  that  every  pixel  provides  two  constraint  equations  which  are  linear  in  the  spline 
coefficients.  Refering  to  Eq.2,  we  have  Kf  ■  Kg  spline  coefficients  that  need  to  be  estimated.  We 
therfore  need  to  use  constraints  at  least  Kf^9  pixels  to  linearly  solve  for  the  coefficients.  For 
accurate  mirror  shape  computation  we  typically  use  every  image  pixel.  We  then  estimate  spline 
coefficients  by  solving  an  overdetermined  system  of  equations  formed  by  stacking  together  all  the 
constraints. 

It  should  be  noted  that  to  solving  for  the  mirror  shape  requires  knowing  the  mirror  surface  normal 
at  every  sampled  image  pixel.  Referring  to  Fig.  4,  we  observe  that  the  mirror  can  lie  anywhere 
within  the  field  of  view  of  the  primary  optics.  The  two  mirrors  S/1-*  and  Sr^  in  Fig.  4  lie  at 
different  locations  and  must  reflect  the  same  scene  point  A 4(u,v)  into  the  primary  optics.  The 
direction  Vr(«,  v)  along  which  the  scene  point  A4(u,  v )  is  viewed  depends  on  the  mirror  location. 
This  in  turn  influences  the  surface  normals  on  the  mirror  and  hence  its  shape. 

We  resolve  this  cyclical  dependency  using  an  iterative  method.  We  first  approximate  the  mirror 
by  a  set  of  facets  (say,  one  for  each  pixel)  that  lie  on  a  plane.  Only  the  distance  of  the  plane  from 
the  primary  optics  is  chosen  by  the  designer.  We  then  estimate  the  initial  set  of  surface  normals 
using  the  reflected  viewing  direction  Vr  and  incident  ray  from  the  primary  optics  Vi.  These 
normals  are  used  to  linearly  solve  for  the  mirror  shape.  We  now  iterate  by  using  the  computed 
mirror  shape  in  each  iteration  to  estimate  a  better  estimate  of  the  surface  normals  for  the  next 
iteration.  Typically,  convergence  is  achieved  within  10  iterations.  An  example  of  how  the  mirror 
shape  evolves  from  the  initial  planar  guess  to  its  final  shape  is  shown  in  Fig.  5.  In  most  cases,  the 
mirror  shape  is  close  to  its  final  shape  after  just  one  iteration. 

Note  that  the  iterative  approach  is  required  only  for  finte  depth  scenes.  When  the  scene  can  be 
assumed  to  be  very  distant  (at  infinity),  only  viewing  directions  need  to  be  specified.  In  this 
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Figure  4:  The  direction  along  which  a  prescribed  scene  point  A i(u,  v)  is  viewed  by  the  sensor  depends  on 
the  position  of  the  point  of  reflection  on  the  mirror.  This  is  particularly  true  for  close  by  scenes  rather 
than  distant  scenes.  As  shown  here,  changing  the  location  from  S^0)  to  S^1)  alters  the  reflection  direction, 
thus  changing  the  surface  normal. 

setting  the  required  mirror’s  surface  normals  do  not  change  since  the  incident  rays  (from  primary 
optics)  and  reflected  rays  (viewing  directions)  remain  constant,  and  the  mirror  shape  is  computed 
directly  in  one  iteration  alone. 


5  Example  Mirror  Designs 

We  now  use  our  method  to  compute  mirror  shapes  for  different  catadioptric  imaging  and  projection 
systems.  These  include  previously  designed  systems  as  well  as  two  new  ones.  The  previously 
designed  systems  we  present  include  the  parabolic  reflector  based  single  viewpoint  system  [19, 
l],  the  equi-angular  system  [6],  the  plane  rectifying  system  [14].  We  also  present  a  recently 
proposed  panoramic  imaging  system  [l3].The  new  systems  we  present  include  an  imaging  system 
for  teleconferencing  applications  and  a  catadioptric  spherical  emmersive  display  projector  design. 
Specially  in  the  last  two  cases,  no  mirror  exists  that  can  provide  the  required  image  warping. 
We  compare  mirror  shapes  computed  using  the  traditional  mirror  shape  based  approach  to  our 
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Iteration:  5  Iteration:  10  (Final  iteration) 


Figure  5:  Convergence  of  the  mirror  shape  designed  for  the  skewed  plane  projection  in  Fig.  9.  (a)  Initial 
planar  guess  for  the  mirror  with  facets  having  different  surface  normals,  (b)  After  the  first  iteration,  the 
mirror  shape  is  already  close  to  the  final  solution,  (c)  After  5  iterations,  the  mirror  changes  a  negligible 
amount  indicating  fast  convergence,  (d)  The  final  mirror  shape  obtained  after  10  iterations. 

image  error  minimizing  approach.  We  also  present  the  viewpoint  loci  (cautsics)  for  the  computed 
catadioptric  systems.  A  detailed  explanation  of  computing  caustics  for  the  spline  based  mirrors 
can  be  found  in  [24], 

5.1  Parabolic  Mirror  based  Single  Viewpoint  System 

We  computed  the  mirror  for  a  para-catadioptric  [19]  imaging  system  using  a  telecentric  lens  for  the 
primary  optics  (orthographic  projection).  The  theoretical  mirror  shape  for  this  system  is  known 
to  be  parabolic.  The  mirror  was  designed  by  specifying  an  image-to-scene  map  such  that  all  the 
imaged  rays  pass  through  a  virtual  viewpoint  located  1  inch  below  the  apex  of  the  reflector. 

As  seen  in  Fig.  6(a),  the  profile  of  the  computed  mirror  agrees  with  the  analytic  parabolic  profile. 
We  show  the  three-dimensional  shape  of  the  computed  mirror  in  Fig.  6(b).  The  errors  in  the 
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(a)  Comparison  between  computed  and  ideal  mirror 


(b)  Computed  mirror  in  3D 


(c)  Image  errors  (view  angle)  in  degrees  (d)  Caustic  of  the  imaging  system 


Figure  6:  Results  of  applying  the  mirror  design  method  for  the  para-catadioptric  imaging  system  [19] . 
(a)  The  mirror  shape  computed  using  the  proposed  method  is  identical  to  thlo  le  ground  truth  solution 
(parabolic),  (b)  The  3D  shape  of  the  mirror  shown  for  visualization,  (c)  The  mirror  design  being  exactly 
equal  to  the  ground  truth  produced  no  image  projection  error,  (d)  The  caustic  is  a  very  compact  point 
cloud  (single  viewpoint),  as  expected. 


recovered  mirror  shape,  in  terms  of  the  viewing  angles,  are  practically  zero.  The  small  errors  we 
observed  (on  the  order  of  10-8)  were  due  to  numerical  imprecision.  Finally,  the  caustic  surface 
(viewpoint  locus)  for  this  system  was  computed  and  found  to  be  a  very  compact  cluster  of  points 
(essentially  a  single  viewpoint,  as  expected). 


5.2  Equi-Angular  Imaging  System 

We  now  present  results  for  the  equiangular  imaging  system  developed  by  Chahl  and  Srinivasan  [6]. 
In  [6],  a  family  of  mirror  surfaces  that  maintain  a  linear  map  between  scene  rays  and  reflected 
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(a)  Comparison  between  computed  and  ideal  mirror 


Z 

n 


(in  inches) 


(b)  Computed  mirror  in  3D 


(c)  Image  errors  (view  angle)  in  degrees  (d)  Caustic  of  the  imaging  system 


Figure  7:  Results  of  applying  the  proposed  mirror  design  method  for  the  equi- angular  imaging  system  [6]. 
(a)  The  computed  mirror  shape  is  identical  to  the  ground  truth  (see  Eq.(23)).  (b)  The  three  dimensional 
shape  of  the  computed  mirror,  (c)  The  mirror  design  being  an  close  fit  to  ground  truth,  produces 
negligible  viewing  direction  errors,  (d)  The  computed  caustic  surface. 


rays  entering  the  primary  optics  were  derived  as: 

cos  (9  (1  +  /3)  /2)  =  (r/r0)“(1+/3)/2  ,  (22) 

where,  r2  =  x2  +  y2,  9  =  tan ~l{y/x)  and  j3  controls  the  angular  magnification. 

We  tested  our  method  using  various  values  of  /3  (/3  =  3,5,7).  In  all  cases  the  computed  mirror 
shape  was  identical  to  ground  truth.  We  present  results  for  the  case  /3  =  5,  for  which  the  mirror 
profile  (ground  truth)  is  given  by: 

x  ( x 2  —  3 y2)  =  x q.  (23) 

As  shown  in  Fig.  7(a),  the  plot  of  the  computed  mirror  closely  matches  the  ground  truth  shape 
(see  Eq.(23)).  The  errors  in  viewing  direction  (scene  ray)  across  the  image  plotted  in  degrees 
are  negligible  as  seen  in  Fig.  7(c).  The  regular  pattern  in  the  error  plot  is  possibly  due  to  the 
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(a)  Setup  for  cylindrical  panorama  system 


(b)  Computed  mirror  in  3D 


F(pixels) 


Max  Error  =  0.65°  RMS  Error  =  0.25° 


(c)  Image  errors  (view  angle)  in  degrees 


Z 


Figure  8:  Results  for  the  cylindrical  panoramic  imaging  system  [13,  21].  (a)  The  setup  shows  a  mirror 
reflecting  the  scene  around  the  system  into  the  primary  optics  to  directly  form  a  cylindrical  panorama 
in  the  image,  (b)  The  computed  mirror  shape  closely  resembles  those  obtained  previously  [13]  -  (c) 
The  computed  mirror  exhibits  small  errors  in  the  viewing  directions  across  the  image,  (d)  Again,  the 
computed  caustic  of  this  complex  catadioptric  system  was  not  known  until  now. 


piece-wise  nature  of  the  spline  model,  used  for  the  mirror.  This  catadioptric  system  does  not  have 
a  single  viewpoint.  The  computed  caustic  surface  is  shown  in  Fig.  7(d). 


5.3  Cylindrical  Panorama  Imaging  System 

Most  omnidirectional  catadioptric  cameras  acquire  a  panoramic  image  that  is  distorted  and  re¬ 
quires  computational  resources  to  unwarp  into  a  cylindrical  panorama.  Hicks  [13]  and  Srini- 
vasan  [2l]  independently  presented  a  design  of  a  catadioptric  camera  that  directly  acquires  a 
cylindrical  panoramic  image.  Here,  we  design  the  same  system  using  our  method. 

Fig.  8(a)  shows  a  schematic  of  the  desired  imaging  system.  A  mirror  reflects  the  scene  into  the 
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primary  optics  consisting  of  a  telecentric  lens  to  form  a  cylindrical  panorama  in  the  image.  Each 
image  pixel  (it,  v )  is  mapped  directly  to  the  desired  viewing  direction  Vr(it,  v)  (assuming  the  scene 
to  be  at  infinity).  The  horizontal  field  of  view  is  360  degrees,  while  the  vertical  field  of  view  is  60 
degrees. 

Since  the  scene  is  assumed  to  lie  at  infinity,  it  is  possible  to  compute  the  mirror  shape  using  a 
single  iteration  of  our  method.  We  computed  the  mirror,  shown  in  Fig.  8(b),  for  an  image  of  aspect 
ratio  575  :  100.  Note  that,  only  for  certain  aspect  ratios  of  the  image  does  this  image-to-scene 
map  have  an  exact  mirror  solution.  For  the  aspect  ratio  we  chose,  no  exact  mirror  solution  exists. 
Yet,  the  errors  in  the  viewing  directions  are  small  (see  Fig.  8(c)).  The  caustic  of  this  complex 
imaging  system,  shown  in  Fig.  8(d),  was  unknown  until  now. 

5.4  Skewed  Plane  Projection  System 

We  now  present  a  catadioptric  projection  system  geared  towards  avoiding  occlusions.  In  typical 
frontal  projection  systems,  the  display  surface  is  often  occluded  when  the  user  comes  between  the 
projector  and  the  display  surface.  Techniques  to  reduce  occlusions  [22]  work  to  a  limited  extend 
and  produce  artifacts  at  the  shadow  edges. 

We  propose  to  minimize  occlusions  by  positioning  the  projector  very  close  to  the  display  surface. 
However,  in  such  a  configuration,  the  image  needs  to  be  warped  prior  to  projection.  Digitally 
warping  the  image  leads  to  image  quality  degradation  due  to  re-sampling.  Furthermore,  projecting 
images  onto  large  areas  from  such  close  proximity  to  the  display  surface  is  impossible  using 
conventional  projectors  due  to  their  restricted  field  of  view.  We  therefore  propose  optically  warping 
the  image  using  catadioptrics. 

Referring  to  Fig.  9  (a),  let  us  assume  that  the  display  surface  lies  1  foot  below  and  1  foot  away  from 
the  projector  and  spans  a  10'  x  10'  square  region.  The  projector’s  image  plane  is  assumed  to  be 
parallel  to  the  YY-plane.  The  primary  optics  of  the  catadioptric  projection  system  are  assumed 
to  be  orthographic.  The  computed  mirror  shape  using  the  weighted  method,  its  associated  scene 
projection  errors  and  caustic  are  shown  in  Fig.  9  (b,c,d),  respectively.  As  expected  from  the  setup, 
the  mirror  is  symmetric  about  a  single  plane.  Note  that,  in  spite  of  the  projector  being  only  a  foot 
away  from  a  large  display  surface,  the  errors  in  projection  are  negligible  (less  than  0.02  inches). 
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(c)  Scene  projection  errors  (in  inches) 


(d)  Caustic  of  projection  system 


Figure  9:  A  novel  catadioptric  projection  system  designed  using  the  proposed  framework.  We  design 
a  projection  system  that  minimizes  occlusions  by  placing  it  very  close  to  the  display  surface,  (a)  The 
projector  is  located  1'  above  and  1'  away  from  a  10'  x  10'  display  surface,  (b)  The  computed  three- 
dimensional  mirror  shape,  (c)  Projection  errors,  measured  on  the  display  surface  and  are  negligible 
(<  0.02').  (d)  The  projection  system  does  not  have  a  single  point  of  projection  but  rather  a  caustic, 
computed  using  the  proposed  framework. 


5.5  Conference  Table  Rectifying  System 

We  now  present  a  novel  catadioptric  imaging  system  for  video  conferencing  applications.  Consider 
the  scenario  of  imaging  people  seated  around  an  elliptical  conference  table.  We  would  like  to 
acquire  an  image  such  that  the  curved  edge  of  the  table  appears  straight.  Thus,  all  the  people 
would  appear  as  if  seated  at  a  straight  desk. 

We  call  such  an  imaging  system  as  the  conference  table  rectifying  system.  The  setup  of  the  mirror 
and  table  for  this  system  are  shown  in  Fig.  10(a).  The  table  consists  of  a  semi-circular  section 
of  radius  30  inches  with  two  extended  straight  sides,  each  30  inches  long.  The  primary  optics 
(telecentric  lens)  is  assumed  to  be  located  roughly  30  inches  behind  this  table  facing  away  from 
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(a)  Schematic  of  conference  table  rectifying  system 


(b)  Computed  mirror  in  3D 


10  40  70  100 


Max  error  =  4.25  pixels  RMS  error  =  1.39  pixels 
(c)  Image  projection  errors  (in  pixels) 


Figure  10:  A  new  imaging  system  for  video  conferencing  applications,  (a)  Setup  of  the  imaging  system 
designed  to  acquire  images  of  people  seated  around  a  curved  conference  table  as  if  seated  along  a  straight 
bench.  We  use  a  telecentric  lens  based  primary  optics,  (b)  The  computed  “optimal”  mirror  shape  in 
three  dimensions,  (c)  Errors  in  the  image  projection  are  higher  in  this  case  as  no  mirror  that  performs 
the  desired  map  exists,  (d)  The  corresponding  caustic  surface  is  also  computed.  Since  the  mirror  is 
flatter  along  one  direction,  the  caustic  is  almost  linear  along  that  direction. 


the  scene  into  a  mirror. 

The  computed  mirror  shape  is  shown  in  Fig.  10(b).  Note  that  no  mirror  shape  exists  that  provides 
the  required  image-to-scene  map.  The  computed  mirror  shape  approximates  the  desired  mapping 
by  minimizing  image  distortions.  In  this  case,  it  was  therefore  important  to  use  the  modified  image 
error  based  metric  discussed  in  Section3.  The  resulting  image  errors  are  shown  in  Fig.  10(c).  This 
imaging  system  also  does  not  have  a  single  viewpoint.  Its  caustic  is  shown  in  Fig.  10(d). 

We  also  compare  our  method  against  an  hypothetical  ideal  sensor  and  previous  shape  based 
methods.  While  the  previous  method[25]  produces  large  image  errors  (Max.  =  10.14,  RMS  = 
3.0)  (see  Fig.  10(e)),  our  approach  drastically  reduces  errors  by  about  80%  (Max.  =  1.92,  RMS 
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Figure  11:  A  checker  board  pattern  acquired  using  (a)  a  hypothetically  ideal  system,  (b)  a  mirror  designed 
to  minimize  errors  in  orientation  of  the  surface  normals  [l  1 ,  25],  showing  strong  distortions  (Max.  = 
10.14,  RMS  =  3.0)  and  (c)  our  method  that  minimizes  image  errors  (Max.  =  1.92,  RMS  =  0.6).  Note 
that  our  method  reduces  errors  by  over  80%. 

=  0.6).  This  is  demonstrated  by  the  large  distortions  visible  in  Fig.  11(b)  the  shape  based  spline 
approach,  as  opposed  to  Fig.  11(a)  the  ground  truth  or  Fig.  11(c)  our  proposed  approach. 

5.6  Spherical  Projection  System 

Our  last  example  consists  of  a  novel  projection  system  for  an  immersive  viewing  environment. 
The  setup  consists  of  a  spherical  projection  screen  of  radius  10'.  The  observer  is  located  at  the 
center  of  the  sphere,  while  the  projector  is  located  directly  above  the  observer  at  the  pole  of  the 
sphere  as  in  Fig. 12(a).  The  goal  is  to  project  a  section  of  a  spherical  panorama  from  the  pole,  as 
if  projected  from  the  center. 

For  a  given  field  of  view,  Figs.l2(b,c)  illustrate  the  computed  mirror  shape  and  the  corresponding 
scene  errors.  Also,  we  compare  errors  in  projection  due  to  earlier  shape  based  approaches  (Max 
=  12.65,  RMS  =  4.8  inches)  shown  in  Fig. 12(e)  against  that  using  our  proposed  scene  error  based 
method  (Max  =  8.43,  RMS  =  2.84)  (see  Fig. 12(f)).  Projection  errors  using  our  method  were 
reduced  by  over  30%. 

A  point  to  note  is  that  the  same  method  was  used  to  estimate  all  the  mirror  shapes.  We  can 
compute  single  viewpoint  as  well  as  non-single  viewpoint  systems  just  as  easily.  This  makes  our 
method  truly  flexible  and  applicable  to  a  large  class  of  mirror  design  problems. 
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(a)  Spherical  projection  system 


X 


(b)  Computed  mirror  in  3D 


Max  error=8.4"  RMS  error  =  2.84" 
(c)  Scene  projection  errors  (inches) 
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(d)  Ideal  checkerboard  pattern 


(e)  Distorted  pattern  using  previous [21]  tool 


(f)  Checkerboard  with  proposed  method 


Figure  12:  (a)  A  spherical  immersive  screen  with  the  observer  seated  in  the  center.  The  projector  is 
located  on  the  roof  and  is  expected  to  project  a  section  of  a  spherical  panorama  on  the  screen  as  shown, 
(b)  The  mirror  computed  using  our  scene  projection  error  minimizing  approach,  (c)  Errors  in  projection 
onto  the  sphere  are  very  small  in  comparison  with  the  mirror  shape  error  minimizing  technique,  (d) 
Ideal  checkerboard  pattern  projected  on  the  sphere,  (e)  Equivalent  checkerboard  pattern  using  the 
mirror  shape  error  minimizing  method  showing  stronger  distortions,  (f)  Checkerboard  pattern  projected 
using  our  proposed  method  is  visibly  much  lesser  distorted. 

6  Conclusions 


In  this  paper,  we  presented  a  framework  to  design  general  catadioptric  imaging  and  projection 
systems.  Such  systems  are  assumed  to  consist  of  some  known  primary  optics  and  an  unknown 
mirror.  Using  our  framework,  the  designer  has  complete  freedom  in  designing  arbitrary  catadiop¬ 
tric  systems.  One  can  define  any  desired  imaging  geometry,  by  specifying  a  map  from  pixels  in 
the  image  to  points  in  the  scene,  which  we  called  the  image-to-scene  map.  Our  method  then 
automatically  computes  the  optimal  mirror  shape  that  implements  the  desired  image-to-scene 
map. 

A  major  advantage  of  the  method  is  that  it  can  be  used  with  all  possible  models  for  the  primary 
optics.  These  include  the  perspective,  orthographic  and  the  generalized  imaging  model.  Further¬ 
more,  the  same  method  can  be  used  to  design  a  very  large  class  of  systems  including  rotationally 
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symmetric,  asymmetric,  single  viewpoint  and  non-single  viewpoint  systems.  We  also  used  our 
framework  to  compute  the  caustics  of  such  general  imaging  systems. 

We  believe  that  the  method  we  proposed  is  a  powerful  tool  and  throws  wide  open  the  space  of 
projector  and  imaging  systems  that  can  be  designed. 
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